I'm not entirely clear on which problem we're trying to solve here, and I think there are problems in this area we shouldn't try to solve (but instead make sure users have the tools to solve them, which I think they already do).

With the exception of libpython.so (which I'll get back to), we shouldn't have any shared library dependencies (as fulfilled by ld.so) within the Python installation, right? The only shared library we build that things explicitly link to is libpython.so. Any dependencies we (optionally) include in our build (libffi, expat, etc) are linked statically into the extension or binary instead, and all extension modules, while .so files, are built without a dependency on any other .so files inside the Python installation.

Extension modules are .so's, but we don't (and mustn't!) rely on ld.so to find them. We load them explicitly with dlopen(). They can be moved around without problems.

libpython.so is a different matter. When building with --enable-shared, the python binary does depend on libpython.so, and it does use ld.so to load it (after installation). In this case we do rely on ld.so to find libpython, which is a problem when installing in a prefix that isn't in the global shared library search path, and when having multiple different installations of the same python. I think the proper solution here is to not dynamically link python. We can still build the shared library for embedding purposes, but have the python binary instead statically link it. (I'll also just mention that this gives you a significant performance boost, too. Most of that can be recouped with the right compiler and compiler options, but statically linking is much easier to get right.)

(There's a side topic about what to do with third-party extension modules that do link against libpython, or third-party .so's that are expected to be used either as plugins in an embedded Python or by extension modules loaded into a statically linked python. Technically that's a problem if they are loaded in a statically linked python, but in practice, I don't think it matters. If it does matter perhaps we could have an empty libpython.so (or one with trampoline functions?) that Python dlopen()s to satisfy DT_NEEDED. But these cases are already problematic with any python installation that does not use --enable-shared. FWIW, Debian/Ubuntu has been statically linking python and shipping libpython.so for a very long time now. I don't know if it's ever been a problem.)

That takes care of all shared libraries in a Python installation, right? Am I forgetting any?

So when statically linking, the Python installation itself should be just as relocatable as when fiddling with RPATH. (There are probably many other issues to solve, like all the hardcoded paths in sysconfigdata, but that's not fixed by this PR either.)

The other concern seems to be external dependencies: building a Python that uses, for example, a specific version of a shared library that's installed elsewhere. I'm not convinced we should try to solve that in Python itself. It's not specific to how Python is built. It's a property of the larger system you're setting up. Other binaries builtin within such a system have to do the same thing. ELF isn't really set up for this, and it's very easy to end up with a mess of shared libraries loaded from different places. (I know, I worked at Google on their version of this, which involves separate ld.so's and libc's in alternative places, and it still messed up every now and then because of, for example, config files living in /etc despite the install directory of the library being elsewhere.)

Moreover, if they're dependencies of extension modules, the building of the extension module can include setting RPATH, without affecting the python binary. ($ORIGIN in that case is relative to the .so.) You can even do that for individual extension modules in the standard library, by editing Modules/Setup.

If we try to solve this problem for the Python binary we may just end up creating an attractive nuisance, and then actively hampering other approaches to solving the same problem. For example, this PR adds $prefix/lib to the search path, but someone building a self-contained system like this may need other paths to be added. Or they may want to move the library somewhere else, or move the python binary out of the bin directory (which is fine for Python as long as there's a lib sub-directory containing the python3.14 subdirectory somewhere in a parent directory). I'm afraid setting RPATH to include the directory containing libpython.so won't actually help this use-case. And why does this need to be a configure flag? Why can't users not just add the options to CFLAGS? This gives them much more flexibility.

Pablo and I had a chat, and he's convinced me that users don't have the tools to solve the second problem. We need a better way to pass linker flags to extension modules that's separate from passing linker flags to the python binary. I think we agreed that this PR isn't it, though, since it only works for --rpath.

Moreover, if they're dependencies of extension modules, the building of the extension module can include setting RPATH, without affecting the python binary. ($ORIGIN in that case is relative to the .so.) You can even do that for individual extension modules in the standard library, by editing Modules/Setup.

As a side note, since 3.11, configure supports supplying unique CFLAGS and LDFLAGS options for each third-party library used by standard library extension modules, potentially obviating the need to edit Modules/Setup:

https://docs.python.org/dev/using/configure.html#options-for-third-party-dependencies

As a side note, since 3.11, configure supports supplying unique CFLAGS and LDFLAGS options for each third-party library used by standard library extension modules, potentially obviating the need to edit Modules/Setup:

It supports CFLAGS but not LDFLAGS no?

We are also missing a global option

Closing as per above

It supports CFLAGS but not LDFLAGS no?

AFAIK, while the configure options are named *_LIBS, their values are passed to the linking commands just like LDFLAGS.

It supports CFLAGS but not LDFLAGS no?

AFAIK, while the configure options are named *_LIBS, their values are passed to the linking commands just like LDFLAGS.

Checked the docs and indeed it's documented as:

.. option:: BZIP2_CFLAGS
.. option:: BZIP2_LIBS

   C compiler and linker flags to link Python to ``libbz2``, used by :mod:`bz2`
   module, overriding ``pkg-config``.

this is quite confusing but you are right 👍

In any case we are still missing global options for all of them